Throughout this disclosure including in the claims, the term “processor” is used in a broad sense to denote a system or device programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chip set), a digital signal processor programmed and/or otherwise configured to perform pipelined processing on audio or other sound data, a programmable general purpose processor or computer, and a programmable microprocessor chip or chip set.
Throughout this disclosure including in the claims, the expressions “audio processor” and “audio processing unit” are used interchangeably, and in a broad sense, to denote a processor configured to process audio data. Examples of audio processing units include, but are not limited to encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems. Examples of audio processors include appropriately configured field-programmable gate arrays (or other configurable integrated circuits or chip sets), digital signal processors programmed and/or otherwise configured to perform pipelined processing on audio or other sound data, appropriately programmed (and/or otherwise configured) programmable general purpose processors or computers, and appropriately programmed (and/or otherwise configured) programmable microprocessor chips or chip sets.
Herein, the expressions “audio processing methods” and “audio processing algorithms” are used interchangeably, as synonyms, to denote methods implemented by audio processors, which process input audio data to generate output (processed) audio data in response to the input audio data.
A wide variety of audio processing algorithms are in commercial use to perform various types of useful processing on audio data.
It is often useful to perform benchmarking of an audio processing algorithm (“APA”), to determine a benchmark indicative of power consumption and optionally also at least one other performance characteristic (e.g., cycle complexity) of the APA as executed by (i.e., as running on) an audio processor configured to execute the APA.
Power and performance (e.g., cycle complexity) measurements of APAs running on modern audio processors, especially portable devices, are both crucially important (since power consumption affects battery life) and very difficult. The inventors have recognized that part of the difficulty stems from the way that modern processors use dynamic voltage and clock frequency scaling to minimize steady-state power consumption while still maximizing peak or burst power consumption. The inventors have further recognized that the existence of these features prevents traditional benchmarks of APAs, which are determined in a benchmarking operation with each node running in a “batch” mode (i.e., as fast as it can until processing of the relevant audio data is finished), from accurately representing the cost (e.g., power consumption) of each of the same APAs running in a “real time” mode (i.e., in the same way, at least in critical respects, that it is expected to run in real deployment.
Throughout this disclosure including in the claims, the expression that APA being benchmarked is executed (or runs) in a “real time mode” in a processor to generate processed audio, denotes that the audio processing is performed in a manner which simulates expected real time execution of the APA by a contemplated deployed system (an audio processing system to be deployed by an end user) which executes the APA to be benchmarked. For example, in some embodiments of the invention, and APA is executed in a run time mode (during benchmarking) in which the audio processing is performed only during processing intervals (each having selected duration) on bursts of audio data (with a burst of audio data undergoing processing in each processing interval) separated by intervals (“sleep intervals”) of selected duration in which audio processing is not performed (each sleep interval occurring between two consecutive processing intervals) in a manner which keeps signal output buffers (which buffer the processed audio) from emptying, and which matches or is intended to match the same way the audio processing is to be performed during expected real time execution of the APA by a deployed system (at least in the respect that its average rate of audio sample processing matches the expected average rate of audio sample processing in real deployment, and optionally also in at least one other respect). Benchmarking of APAs running in a real time mode is intended to cause the resulting benchmarks to be indicative of execution of the APAs in a real audio processing framework, so that each APA being benchmarked is executed (during benchmarking) in the same way (at least in the critical respect that its average rate of audio sample processing matches the expected average rate of audio sample processing in real deployment) that it is expected to be executed in real deployment. Typically, execution in a real time mode of an APA being benchmarked simulates (e.g., by appropriate choice of burst size and sleep interval duration) variation (“throttling”) of CPU voltage and/or frequency by the contemplated deployed system (e.g., throttling up to a maximum and then, if the processing continues for a sufficient duration and sufficiently rapidly that measured processor temperature rises faster than heat can be dissipated, back down to a lower value).
As more audio processing algorithms are developed for execution by mobile devices (e.g., tablets and smartphones) there is a greatly increased sensitivity to the power that products (which are programmed or otherwise configured to execute such algorithms) will consume during execution of the algorithms. This is in contrast to the demands for better and more immersive experiences that demand dramatically more sophisticated and complex audio processing algorithms, at the same time as the target hardware platforms have reached a slowing of the Moore's-law progression of ever-better performance and efficiency. It is currently very important to measure both execution time (e.g., as determined by the processing rate (e.g., in MCPS, or millions of clock cycles per second) and number of clock cycles required to process the relevant audio data) and, more importantly, power consumption of systems (especially mobile devices) which execute audio processing algorithms.
Unfortunately, just as it becomes increasingly important to make such benchmarking measurements, it is also becoming more difficult to do so. The inventors have recognized that one reason for the increasing difficulty is that hardware platforms are increasingly using aggressive processor voltage and clock scaling, which mostly makes the traditional, iterative batch mode of benchmarking inappropriate. Traditionally, an APA undergoing benchmarking would be executed back-to-back a large number of times, so that the execution time of a single iteration could be estimated accurately. On typical modern hardware such processing represents a peak computation load that will typically cause the CPU voltage and frequency to be throttled up to a maximum, and (if the test goes on for long) perhaps throttled back down as the system measures the processor temperature rising faster than the heat can be dissipated. This throttling makes conventional long-time average measurements largely meaningless. To make matters worse, the power consumption of a processor is nonlinear with voltage (and the voltage is usually scaled up to support higher CPU frequencies), because broadly speaking it follows a V2R law. The inventors have recognized that the only way to accurately measure power consumption (and complexity performance) of APAs running in a large number of modern devices is to perform the measurement with the APAs (or synthetic APAs which correspond to the APAs) with the APAs (or corresponding synthetic APAs) running in as close to their expected final, delivered form as possible. They have also recognized that this implies that benchmarking of an APA should (at least in typical circumstances) be performed with the APA (or a synthetic APA corresponding thereto) running in a real time mode which simulates expected real time execution of the APA by a deployed system (e.g., with audio processing in periodically occurring bursts on blocks of audio data, using as little of the processor's resources as possible to match an expected throughput rate in real deployment).
The inventors have also recognized that it is also important that a delivered benchmark of an APA be determined with usage of processing system memory, both in size and access patterns, similar to that expected to occur in expected real deployment of the benchmarked algorithm (e.g., typical intended use of the audio processing code to be delivered to the intended users). Underlying this recognition is their recognition that the relatively large memory space required to store and process many channels of audio data, with selectivity for low frequencies (and therefore with considerable buffering and latency) often causes end user implementations to have worse than the optimal performance achievable in a benchmarking operation (e.g., due to the relatively high cost of cache misses, both in direct power consumed and in processor stall cycles which occur in real deployment).
The inventors have also recognized that it would be desirable to provide and disseminate a system for benchmarking an APA as it would run on each of one or more deployed audio processing systems (or computer-implementable code for programming a system to perform such benchmarking), without actually disseminating the APA. For example, the APA may include secrets so that it is not desired to disseminate the actual APA. In view of this recognition, an aspect of some embodiments of the present invention is a method and system for benchmarking a “synthetic” APA as it would run on each of one or more deployed audio processing systems (or computer-implementable code for programming a system to perform such benchmarking), where the synthetic APA corresponds to and emulates (as described herein) a real (functional) APA, where the functional APA (or variations thereon) is intended for real use (running on a deployed system) and where it is desired to benchmark the functional APA (or a variation thereon) as it would run on the deployed system. Such a synthetic APA may be freely disseminated (e.g., included in computer-implementable code for programming a system to perform an embodiment of the inventive benchmarking method), while the corresponding functional APA is maintained as a secret.